Variance-Reduced Stochastic Learning by Networked Agents under Random Reshuffling

A new amortized variance-reduced gradient (AVRG) algorithm was developed in \cite{ying2017convergence}, which has constant storage requirement in comparison to SAGA and balanced gradient computations in comparison to SVRG. One key advantage of the AVRG strategy is its amenability to decentralized implementations. In this work, we show how AVRG can be extended to the network case where multiple learning agents are assumed to be connected by a graph topology. In this scenario, each agent observes data that is spatially distributed and all agents are only allowed to communicate with direct neighbors. Moreover, the amount of data observed by the individual agents may differ drastically. For such situations, the balanced gradient computation property of AVRG becomes a real advantage in reducing idle time caused by unbalanced local data storage requirements, which is characteristic of other reduced-variance gradient algorithms. The resulting diffusion-AVRG algorithm is shown to have linear convergence to the exact solution, and is much more memory efficient than other alternative algorithms. In addition, we propose a mini-batch strategy to balance the communication and computation efficiency for diffusion-AVRG. When a proper batch size is employed, it is observed in simulations that diffusion-AVRG is more computationally efficient than exact diffusion or EXTRA while maintaining almost the same communication efficiency.Comment: 23 pages, 12 figures, submitted for publicatio

Globally Optimal Beamforming Design for Integrated Sensing and Communication Systems

In this paper, we propose a multi-input multi-output (MIMO) beamforming transmit optimization model for joint radar sensing and multi-user communications, where the design of the beamformers is formulated as an optimization problem whose objective is a weighted combination of the sum rate and the Cram\'{e}r-Rao bound (CRB), subject to the transmit power budget constraint. The formulated problem is challenging to obtain a global solution, because the sum rate maximization (SRM) problem itself (even without considering the sensing metric) is known to be NP-hard. In this paper, we propose an efficient global branch-and-bound algorithm for solving the formulated problem based on the McCormick envelope relaxation and the semidefinite relaxation (SDR) technique. The proposed algorithm is guaranteed to find the global solution for the considered problem, and thus serves as an important benchmark for performance evaluation of the existing local or suboptimal algorithms for solving the same problem.Comment: 5 pages, 2 figures, submitted for possible publicatio

Diversity Order Analysis for Quantized Constant Envelope Transmission

Quantized constant envelope (QCE) transmission is a popular and effective technique to reduce the hardware cost and improve the power efficiency of 5G and beyond systems equipped with large antenna arrays. It has been widely observed that the number of quantization levels has a substantial impact on the system performance. This paper aims to quantify the impact of the number of quantization levels on the system performance. Specifically, we consider a downlink single-user multiple-input-single-output (MISO) system with M-phase shift keying (PSK) constellation under the Rayleigh fading channel. We first derive a novel bound on the system symbol error probability (SEP). Based on the derived SEP bound, we characterize the achievable diversity order of the quantized matched filter (MF) precoding strategy. Our results show that full diversity order can be achieved when the number of quantization levels L is greater than the PSK constellation order M, i.e., L>M, only half diversity order is achievable when L=M, and the achievable diversity order is 0 when L<M. Simulation results verify our theoretical analysis.Comment: 9 pages, 3 figures, submitted for possible publicatio

Connecting Multi-modal Contrastive Representations

Multi-modal Contrastive Representation learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project them to a new space and use the data from the overlapping modality B to aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A, B) and (B, C) are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair (A, C). To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. To demonstrate the effectiveness of C-MCR, we connect CLIP and CLAP via texts to derive audio-visual representations, and integrate CLIP and ULIP via images for 3D-language representations. Remarkably, without using any paired data, C-MCR for audio-visual achieves state-of-the-art performance on audio-image retrieval, audio-visual source localization, and counterfactual audio-image recognition tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40.Comment: NeurIPS 202

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Large language models (LLMs) and vision language models (VLMs) demonstrate excellent performance on a wide range of tasks by scaling up parameter counts from O(10^9) to O(10^{12}) levels and further beyond. These large scales make it impossible to adapt and deploy fully specialized models given a task of interest. Parameter-efficient fine-tuning (PEFT) emerges as a promising direction to tackle the adaptation and serving challenges for such large models. We categorize PEFT techniques into two types: intrusive and non-intrusive. Intrusive PEFT techniques directly change a model's internal architecture. Though more flexible, they introduce significant complexities for training and serving. Non-intrusive PEFT techniques leave the internal architecture unchanged and only adapt model-external parameters, such as embeddings for input. In this work, we describe AdaLink as a non-intrusive PEFT technique that achieves competitive performance compared to SoTA intrusive PEFT (LoRA) and full model fine-tuning (FT) on various tasks. We evaluate using both text-only and multimodal tasks, with experiments that account for both parameter-count scaling and training regime (with and without instruction tuning)

Effect of the Strength of Initial Aluminium on the Bonding Properties and Deformation Coordination of Ti/Al Composite Sheets by the Cold Roll Bonding Process

Ti/Al composite sheets were prepared using the cold rolling process, and different initial aluminium strengths were considered. The results showed that the peel strength of the Ti/Al composite sheet increased with the increasing initial strength of aluminium under the same reduction. A higher strength of the initial aluminium corresponds to better deformation coordination between titanium and aluminium, where the strain hardening of titanium and aluminium plays an important role. The change degree of the components of twins on the titanium side for the Ti/Al composite sheet with a low aluminium strength is stronger than that for the Ti/Al composite sheet with a high aluminium strength. The strong change in the components of twins may result in the low uniformity of the microstructure on the titanium side. The analysis of the peeling surface shows aluminium residue on the titanium side, while there was almost no titanium residue on the aluminium side. At the same reduction, a higher strength of aluminium corresponds to less aluminium residue on the titanium side. The bonding properties of Ti/Al cold-rolled composite sheets were determined by four strong bonding areas. The strength of the initial aluminium was the main factor, and the residual amount of aluminium on the titanium side of the peeling surfaces was a secondary factor